An Introduction to Functional Derivatives

This tutorial on functional derivatives focuses on Fréchet derivatives, a subtopic of functional analysis and of the calculus of variations. The reader is assumed to have experience with real analysis. Definitions and properties are discussed, and examples with functional Bregman divergence illustrate how to work with the Fréchet derivative. 1 Functional Derivatives Generalize the Vector Gradient Consider a function f defined over vectors such that f : R → R. The gradient∇f = { ∂f ∂x1 , ∂f ∂x2 , . . . , ∂f ∂xd } describes the instantaneous vector direction in which the function changes the most. The gradient ∇f(x0) at x0 ∈ R tells you that if you are starting at x0 which direction would lead to the greatest instantaneous change in f . The inner product (dot product) ∇f(x0) y for y ∈ R gives the directional derivative (how much f instantaneously changes) of f at x0 in the direction defined by the vector y. One generalization of a gradient is the Jacobian, which is the matrix of derivatives for a function that map vectors to vectors (f : R → R). In this tutorial we consider the generalization of the gradient to functions that map functions to scalars; such functions are called functionals. For example let a functional φ be defined over over the convex set of functions, G = {g : R → R s. t. ∫ x g(x)dx = 1, and g(x) ≥ 0 for all x}. (1) An example functional defined on this set is the entropy: φ : G → R where φ(g) = − ∫ x g(x) ln g(x)dx for g ∈ G. In this tutorial we will consider functional derivatives, which are analogs of vector gradients. We will focus on the Fréchet derivative, which can be used to answer questions like, “What function g will maximize φ(g)?” First we will introduce the Fréchet derivative, then discuss higher-order derivatives and some basic properties, and note optimality conditions useful for optimizing functionals. This material will require a familiarity with measure theory that can be found in any standard measure theory text or garnered from the informal measure theory tutorial by Gupta [1]. In Section 3 we illustrate the functional derivative with the definition and properties of the functional Bregman divergence [2]. Readers may find it useful to prove these properties for themselves as an exercise. 2 Fréchet Derivative Let ( R,Ω, ν ) be a measure space, where ν is a Borel measure, d is a positive integer, and define the set of functions A = {a ∈ L(ν) subject to a : R → R} where 1 ≤ p ≤ ∞. The functional ψ : L(ν)→ R is linear and continuous if 1. ψ[ωa1 + a2] = ωψ[a1] + ψ[a2] for any a1, a2 ∈ L(ν) and any real number ω 2. there is a constant C such that |ψ[a]| ≤ C‖a‖ for all a ∈ L(ν).